Stop slow peers from blocking healthy traffic - #172

Problem

The peer health loop holds one global connection-map mutex while it dials an unreachable peer. Every other peer waits for that machine.

A retained 358.498-second window measured 300 direct hetz-to-Silber probes. All passed, but 19 exceeded two seconds.

Eighteen slow probes aligned with the 20-second health loop. External p50 was 45.013 ms, p95 was 2.117 seconds, and p99 was 2.675 seconds.

The synchronized internal window recorded 18 Silber probes below 200 ms. It recorded no loss or retry, so that instrument misses the lock wait.

Change

Use one connection mutation gate per peer. Release the global map before the network dial and generation exchange.

The same-peer gate preserves connection replacement and simultaneous-open ordering. A slow peer no longer holds the map used by healthy peers.

The health loop remains sequential. That can delay later health samples, but it cannot delay application traffic after this change.

Proof

The new deterministic test pauses one peer during generation exchange. It then opens a stream through an existing connection to another peer.

The test failed on current main in 0.73 seconds. It passed after the change.

• cargo test mux::tests -- --nocapture: 14 passed and one measurement ignored.
• cargo test --lib: 514 passed and five measurements ignored.
• The simultaneous-peer integration test passed.

The retained production files remain on hetz under /tmp/fabric-latency-1788586961-*. The telemetry blindness needs a separate fix.